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1 . SCOPE 


a.  This  document  presents  methodology  which  should  be  followed 
by  system  analysts  and  simulation  personnel  in  the  development  and 
application  of  simulation  programs  to  support  system  test  and  evalua- 
tion missions.  Simulation  development  is  presented  as  a succession 
of  the  following  five  closely  related  and  often  iterative  stages: 

(1)  System  analysis  and  model  requirements  definition 


(2)  Implementation 

(3)  Verification 

(4)  Validation 

(5)  Project  applications 


The  objectives  for  each  of  the  fi 
and  the  analytical  and  investigative  procedures  for  accomplishing 
those  objectives  are  specified.  Requirements  for  project  documenta- 
tion for  each  stage  of  simulation  development  are  also  presented. 


development  stages  are  detailed. 
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b.  This  TOP  addresses  the  need  for  acceptance  of  standard 
terminology  and  methodology  to  be  used  in  development  activities 
across  the  simulation  community,  and  the  five-phase  program 
presented  in  section  4 is  recommended  as  a standard  approach. 

c.  The  detail  to  which  this  TOP  should  be  applied  depends  on 
the  size  and  complexity  of  the  desired  simulation. 

2.  INTRODUCTION 


a.  Simulation,  as  an  aid  to  design,  development,  and  test  and 
evaluation  of  complex  Army  systems,  is  an  essential  resource  which 
affords  a means  whereby  analysts  may  examine  system  performance 
responses  when  actual  system  testing  either  cannot  be  performed  or 
is  impractical.  During  the  system  design  and  development  stages, 
simulation  can  be  used  to  assess  the  impact  of  design  and  implementa- 
tion plans  on  system  performance  objectives.  During  system  test  and 
evaluation,  simulation  can  be  used  to  obtain  data  for  making  perfor- 
mance assessments  when  resource  constraints  and  the  complexity  of 
the  sy t m itself  make  it  impractical  and  often  impossible  to  demon- 
strate performance  objectives  by  exhaustive  testing. 

b.  For  simulation  to  be  as  effective  as  possible  in  supporting  test 
and  evaluation  objectives,  it  is  essential  that  a well-defined  methodology 
be  established  and  maintained  — a methodology  which  requires  close  coor- 
dination among  simulation  personnel,  system  developers,  system 
analysts,  and  system  test  and  evaluation  directors.  This  methodology 
involves  thorough  planning  for  model  and  simulation  development  and 

for  system  test  design  which  will  accommodate  simulation  validation. 

c.  The  intent  of  this  document  is  to  describe  the  methodology 
which  should  be  established  by  simulation  groups  to  effectively  support 
system  test  and  evaluation.  However,  the  procedures  discussed  are 
commonly  applicable  for  simulation  development  in  support  of  all 
system  investigation  activities  (design,  development,  test  and  eval- 
uation, capability  assessment,  etc.  ).  Analogous  to  Mihram's 

(1)  discussion,  this  methodology  is  presented  as  a five-s  tage 
simulation  development  program  as  follows: 

(1)  A thorough  systems  analysis  investigation  should  be  conducted 
jointly  by  simulation  personnel,  test  directors,  and  systems  analysts  for 


1.  Mihram,  G.  Arthur  (1972)  Simulation:  Statistical  Foundations  and 
Methodology,  Academic  Press 
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the  purpose  of  identifying  simulation  requirements.  This  activity  should 
also  identify  the  system  performance  objectives  which  must  be  evaluated 
by  using  simulation. 

(2)  Once  the  detailed  requirements  have  been  established  and 
documented  (model  functional  requirements),  the  implementation  stage 
should  be  initiated.  This  activity  spans  model  development  (development 
of  math  models,  algorithms,  etc.  , to  depict  system  characteristics  and 
relationships)  and  computer  simulation  program  implementation  (coding, 
installation,  and  checkout  when  development  is  required,  modification 
and/or  adaptation  when  existing  programs  or  routines  are  available). 

(3)  Simulation  verification  is  the  third  stage  in  the  development 
cycle,  and  though  it  is  a specific  activity  in  and  of  itself,  it  cannot  be 
divorced  from  any  of  the  other  development  stages.  The  basic  objectives 
of  verification  are  to  establish  the  integrity  of  the  implemented  simula- 
tion with  respect  to  program  design  (system  model)  and  to  establish  the 
integrity  of  the  program  design  with  respect  to  the  system  which  has 
been  modeled.  As  a result,  verification  should  be  a basic  concern 
throughout  initial  system  analyses,  model  design  and  development, 
simulation  program  checkout,  and  project  applications. 

(4)  As  soon  as  a simulation  program  has  been  implemented  and 
checked  out  and  performance  data  from  actual  system/subsystem  tests 
are  available,  simulation  validation  can  begin.  The  intent  for  this 
activity  should  be  to  establish  the  correspondence  between  system 
responses  and  simulation  responses  for  comparable  external  stimuli. 

The  objective  for  validation  investigations  is  threefold: 

(a)  It  serves  to  answer  questions  concerning  adequacy  of  modeling 
detail  which  were  uncovered  during  verification  investigations  by 
demonstrating  acceptable  (unacceptable)  performance: 

(b)  It  identifies  errors  in  program  design  which  often  require  a 
review  of  systems  analyses  for  correction; 

(c)  It  provides  data  for  making  an  assessment  of  the  credibility 
of  the  simulation  and  either  certifies  it  for  or  disqualifies  it  from  use 
in  evaluating  system  performance  in  the  project  applications  stage  to 
follow. 

It  should  further  be  noted  that  simulation  validation  comparisons  should 
be  made  against  the  results  from  system  tests  which  were  carefully 
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designed  to  establish  critical  values  for  the  key  system  performance 
parameters.  While  all  available  system  test  data  can  be  used  for 
simulation  validation,  the  correspondence  with  test  data  from  system 
boundary  tests  does  more  to  establish  simulation  credibility. 

(5)  The  project  applications  stage  of  simulation  development  is 
a realization  of  the  objectives  for  which  the  simulation  development 
was  initiated.  Once  an  acceptable  level  of  confidence  is  developed 
in  the  program's  capacity  to  represent  the  actual  system  or  to  per- 
form as  the  actual  system  would,  the  program  can  then  be  utilized 
for  predicting  system  performance  on  subsequent  tests,  for  further 
test  design,  and  even  for  making  inferences  concerning  system  per- 
formance where  tests  cannot  be  conducted.  This  is  not  to  say  that 
the  simulation  is  perfect  when  it  reaches  this  stage.  Often,  during 
this  phase,  problems  arise  which  require  reiteration  of  the  earlier 
development  stages.  Systems  analyses  may  have  to  be  reexamined, 
the  program  may  have  to  be  modified  or  corrected,  and  verification 
and  validation  may  have  to  be  performed  on  the  revised  program 
before  it  can  again  be  used  for  system  evaluations. 

d.  Throughout  each  of  the  five  development  stages,  careful 
documentation  is  essential.  Well-organized  plans  for  all  investiga- 
tions with  carefully  developed,  realistic  schedules  help  to  insure 
timely  and  effective  results,  and  detailed  specification  and  user 
documentation  promotes  effective  program  utilization  and  helps  to 
deter  redundant  development  efforts  and  program  misapplication. 
Figure  1,  which  was  adapted  from  a figure  by  Mirahm  (Footnote  1), 
depicts  the  order  and  relationships  for  the  five  development  stages 
and  their  documentation  requirements. 

e.  As  discussed  above,  simulation  can  be  a highly  effective  tool 
in  accomplishing  a test  and  evaluation  mission;  however,  everyone 
involved,  from  the  project  manager  to  the  individual  who  codes  a 
simulation  program,  must  agree  on  objectives,  limitations,  and 
requirements  (particularly  validation  requirements).  Above  all, 
complete  cooperation  is  essential. 

3.  DEFINITION  OF  TERMS 


a.  System:  a collection  of  regularly  interacting  or  interdependent 
components  acting  as  a unit  in  carrying  out  an  implicitly  or  explicitly 
defined  mission.  (As  an  example,  an  Army  air  defense  system  might 
be  comprised  of  a number  of  interdependent  components  such  as 
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weapon  groups,  radar  groups,  weapon  control  computers,  and  command 
and  control  groups,  all  of  which  must  interact  to  form  a unit  with  the 
specific  mission  of  detecting,  engaging,  and  destroying  enemy  threats 
according  to  a predefined  doctrine. ) 

b.  System  performance  parameter:  a measurable  system  output 
variable  which  may  be  examined  to  determine  whether  the  system  and/or 
its  components  (subsystems)  are  attaining  performance  levels  consistent 
with  mission  objectives 

c.  System  response:  a measurement  or  a set  of  measurements  on 
any  of  the  system  performance  parameters  triggered  by  external  stimuli 

d.  System  analysis:  an  activity  which  involves  examination  of  a 
system  (either  planned  or  in  existence)  for  the  purpose  of  identifying 
all  system  components  and  the  component  interrelationships  which  are 
essential  to  accomplishing  the  system  mission.  (Of  necessity,  this 
involves  conducting  an  investigation  to  identify  all  external  stimuli  and 
to  determine  how  they  influence  the  system,  to  identify  the  system’s 
processes  and  to  determine  how  they  are  performed,  and  to  identify  the 
system's  performance  parameters  and  to  specify  the  means  for  examining 
system  responses  in  evaluating  the  system.) 

e.  System  model:  a functional  (often  mathematical)  description  of 
a system.  This  functional  description  is  established  for  the  purpose  of 
answering  specific  questions  concerning  the  actual  system's  performance 
and,  depending  on  application  requirements,  the  amount  of  system  detail 
in  models  will  vary  from  problem  to  problem.  However,  the  require- 
ments common  for  all  system  models  are  that  the  expressions  (equations, 
algorithms,  etc.)  in  the  functional  description  must  maintain  the  inter- 
relationships between  all  system  components,  and  that  input-to-output 
transformations  for  the  processes  modelled  must  be  equivalent  to  those 
of  the  actual  system. 

f.  Continuous  system:  a dynamic  system  whose  state  variables 
vary  continuously  as  functions  of  time.  This  type  of  system  can  be 
modelled  using  time-dependent  mathematical  formulae  (differential 
equations,  etc.  ). 

g.  Discrete  event  systems:  those  systems  whose  state  variables 
are  altered  only  at  discrete  instants  of  time.  The  state  variables  may 
be  either  functions  of  time  or  logic  functions  (events  the  occurrence  of 
which  may  depend  on  previous  event  outcomes  and  logic  decision  rules). 
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h.  Computer  program;  either  a set  of  machine-sensible  instruc- 
tions (digital  programs)  arranged  in  a proper  sequence  or  specially 
organized  electronic  circuitry  (analog  program).  Both  analog  and 
digital  programs  are  designed  to  accept  specific  inputs  and  transform 
them  into  required  outputs  (problem  solutions). 

i.  Simulation; 

(1)  A simulation  program  is  a computer  program  (analog,  digital, 
or  hybrid)  which  embodies  a system/subsystem  model  or  models. 

The  program  is  designed  to  accept  external  stimuli  (input  parameters), 
comparable  to  those  which  the  system  would  experience,  to  process 
this  information  and  to  provide  responses  (outputs)  comparable  to  those 
which  would  be  provided  by  the  system. 

(2)  The  term  simulation  is  also  used  when  referring  to  the  total 
five-stage  simulation  development  activity. 

(3)  The  execution  of  a simulation  program  for  a given  set  of 
initial  conditions  (stimuli)  and  the  responses  provided  from  a computer 
run  can  be  called  a simulation. 

j.  Deterministic  variable:  a function  whose  value  is  exactly 
determinable,  given  the  value(s)  of  its  independent  variable(s).  This 
type  of  function,  when  used  to  represent  some  physical  phenomenon, 
is  also  referred  to  as  a deterministic  math  model  of  that  phenomenon. 

k.  Deterministic  process;  a function  of  one  or  more  deterministic 
variables  (functions) 

l.  Random  variable:  a real-valued  function  which  is  associated 
with  some  observable  phenomenon  by  a mapping  rule  which  assigns  a 
real  number  to  an  observation  of  that  phenomenon.  The  values  of  a 
random  variable  may  or  may  not  be  the  same  when  the  phenomenon  is 
observed  repeatedly  under  the  same  conditions. 

m.  Domain  of  a random  variable;  The  set  of  all  possible  observa- 
tions of  a random  phenomenon.  (This  set  is  usually  called  the 
probability  sample  space  for  the  phenomenon.  ) 

n.  Probabilistic  (stochastic)  models;  models  of  phenomena  which 
employ  random  variables. 
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o.  Sample  record  (sample  function):  a single  time  history,  over 
a finite  interval,  representing  measurements  of  some  phenomenon 

p.  Random  (stochastic)  process;  the  totality  of  sample  records 
which  a random  phenomenon  might  produce 

q.  Stochastic  simulation:  a simulation  which  produces  time 
history  records  of  random  phenomena 

r.  Monte  Carlo  simulation:  a technique  whereby  either  random  or 
nonrandom  phenomenon  occurrences  (even  highly  complex,  intractable 
mathematical  function  evaluations)  are  simulated  by  drawing  values 
for  random  variables  from  either  known  or  theorized  probability  dis- 
tributions and  then  solving  mathematical  or  probabilistic  models 

s.  Statistical  hypothesis;  an  assumption  about  the  probability 
density  function  of  a random  variable 

t.  Statistical  hypothesis  teat:  a procedure  for  deciding  whether 
or  not  to  reject  a statistical  hypothesis 

u.  Type  I error:  the  error  of  rejecting  a statistical  hypothesis 
when  it  is  true 

v.  Type  II  error:  the  error  of  not  rejecting  a statistical 
hypothesis  when  it  is  false 

w.  Statistical  inference:  the  process  of  making  decisions  or 
estimations  about  a population  based  on  information  contained  in  a 
sample 


x.  Simulation  verification;  an  investigation  activity  conducted  to 
establish  both  the  integrity  of  the  implemented  simulation  with  respect 
to  program  design  (the  math  model(s)  which  resulted  from  systems 
analysis)  and  the  integrity  of  the  program  design  with  respect  to  the 
system  which  has  been  modelled 

y.  Simulation  validation:  an  investigation  conducted  to  demonstrate 
that  the  simulation  produces  responses  acceptably  comparable  to 
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responses  produced  by  the  actual  system  for  equivalent  stimuli  and 
environmental  conditions.  (This  activity  often  involves  obtaining 
statistical  samples  of  both  simulation  and  system  test  data  and  testing 
the  hypothesis  that  the  samples  belong  to  the  same  population.) 

4.  SIMULATION  METHODOLOGY 


a.  The  discipline  of  modeling  and  simulation  is  one  which  has  been 
expanding  rapidly  into  most  areas  of  the  physical  and  social  sciences. 

It  has  gained  acceptance  as  an  invaluable  tool  for  use  in  systems 
analysis  and  evaluation  by  providing  scientists,  engineers,  systems 
analysts,  and  managers  with  a timely  and  cost-effective  means  of 
gaining  insight  into  how  actual  systems  would  perform  in  specific 
environments  when  testing  is  either  impractical  or  impossible.  It 
follows  that  there  are  as  many  diverse  considerations  in  model  and 
simulation  development  as  there  are  physical  and  social  systems  and 
as  many  as  applications  requirements  dictate.  However,  the  investiga- 
tive procedures  which  must  be  followed  and  the  order  which  must  be 
maintained  in  model  and  simulation  development  remain  invariant 
across  all  disciplines. 

b.  The  intent  of  this  TOP  is  to  present  the  developmental  proce- 
dures and  the  order  of  their  pursuit  as  a conceptual  guide  to  model  and 
simulation  development  which  could  be  followed  in  any  field  of  applica- 
tion. Sections  4.  I through  4.  5 present  detailed  discussions  on  the 
procedures  for  and  requirements  of  each  of  the  five  stages  of  modeling 
and  simulation  development  listed  in  paragraph  la. 

4.  I System  Analysis  and  Model 

Requirements  Definition 

a.  Since  the  fundamental  objective  for  developing  models  and 
simulation  programs  is  to  establish  a capability  which  will  allow 
investigation  of  the  performance  characteristics  of  systems,  either 
planned  or  in  existence,  it  follows  that  the  first  phase  of  development 
should  be  a detailed  study  of  the  system  in  question.  However,  before 
proceeding  with  the  development,  a comprehensive  statement  of  require- 
ment for  simulation  development  must  be  established.  This  is  not  to 
be  confused  with  the  detailed  functional  requirements  of  the  simulation 
to  be  determined  later;  rather,  it  is  a statement  of  -eed  which 
identifies  for  whom  the  simulation  is  to  be  developed  and  which  presents 
the  questions  concerning  system  performance  which  mu3t  be  answered. 
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Without  such  a formalization  of  policy  to  constrain  the  analysis  and 
model  development  activity,  the  program  which  results  may  be  of 
little  or  no  value  in  system  evaluation.  It  should  further  be  emphasized 
that  the  statement  of  requirement  should  be  established  jointly  by 
system  analysts,  system  developers,  simulation  personnel,  and  system 
test  directors  to  preclude  misconception  by  model  developers  of  what 
is  needed  and  misapplication  by  analysts  and  decision  makers  of  what 
the  simulation  will  provide. 

b.  Once  a statement  of  requirement  for  modeling  and  simulation 
development  has  been. formalized,  systems  analysts  and  model 
developers  must  analyze  the  system  (or  at  least  system  design  and 
requirements  documentation)  for  the  purpose  of  developing  detailed 
modeling  functional  requirements.  The  key  investigation  activities 
in  this  effort  involve  studying  the  system  and  its  subsystems  to: 

(1)  Determine  interactions^ 

(2)  Determine  interrelationships 

(3)  Identify  system  and  subsystem  processes  and  their  performance 
requirements 

(4)  Define  the  environment  (inputs,  external  stimuli,  etc.)  within 
which  the  system  must  operate 

(5)  Characterize  system  and  environmental  stochastic  elements 

(6)  Identify  system/subsystem  performance  parameters  (state 
variables,  outputs,  etc.)  which  can  be  examined  to  evaluate  system 
performance 

4.1.1  Analysis  Methodology 

a.  The  methodology  which  should  be  followed  in  the  analysis  and 
requirements  definition  phase  is  a top-down  systems  approach  which 
involves  developing  Hierarchy  plus  Input,  Process,  Output  (HIPO)^ 


2.  Katzon,  Harry  Jr.  (1976)  System  Design  and  Documentation, 
an  Introduction  to  the  HIPO  Method.  Van  Nostrand  Reinhold  Company 
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descriptions  of  the  system  and  its  subsystems.  This  involves  develop- 
ing hierarchical  functional  decomposition  descriptions  of  the  system 
and  its  components  and  processes,  and  developing  complete  descrip- 
tions of  the  input  (environmental  stimuli,  etc.  ) -process  (internal 
functions,  characteristics,  constraints,  etc.  ) -output  (responses  and 
state  variables)  sequence  at  each  level  of  the  system  hierarchy.  At 
the  top  level  of  the  hierarchy,  the  task  involves  the  following: 

(1)  Input  descriptions:  comprehensive  descriptions  of  the  environ- 
ment within  which  the  system  must  operate  and  identification  of  system 
input  parameters  with  complete  characterization  as  to  timing,  units, 
and  possible  parameter  combinations 

(2)  Process  description:  a detailed  functional  description  of  the 
manner  of  accepting  input,  of  the  system  mechanics  for  organizing  the 
actions  of  its  components  (subsystems),  and  of  the  means  whereby 
subsystem  responses  are  organized  to  produce  a system  response.  If 
one  were  to  develop  a hierarchical  diagram  of  the  system's  functional 
decomposition  such  as  the  example  in  Figure  2,  the  system  process 
would  be  the  top  node. 

(3)  Output  description:  a complete  description  of  all  possible  system 
state /response  variables.  There  must  also  be  an  association  of  response 
variables  with  the  possible  combinations  of  input  parameters. 

At  lower  levels  in  the  hierarchy,  the  requirements  for  organizing  analysis 
results  are  the  same  as  those  for  the  system  level.  However,  particular 
attention  must  be  given  to  functionally  depicting  the  interactions  and 
interrelationships  between  the  system  and  its  subsystems  and  between 
subsystems.  It  is  important  to  note  that  each  system  component  (subsystem, 
sub-subsystem,  system  process,  etc.  ) is  also  a system  and  can  be 
described  as  an  input-process-output  sequence.  Subsystem  input  can 
come  from  either  system  external  sources  or  from  sources  strictly 
internal  to  the  system;  however,  the  former  are  consider-  system  input. 
The  subsystem  process  may  be  either  at  a bottom  level  in  the  system 
hierarchy  or  a node  which  combines  activities  of  even  lower-level  system 
components.  Outputs  from  the  subsystem  process  may  be  either  inputs 
to  other  system  processes  or  coordinated  with  outputs  from  other 
processes  to  formulate  a system  response  (output).  Figures  2 and  3 are 
examples  of  the  system's  hierarchical  functional  decomposition, 
depicting  the  input-process -output  sequences. 

b.  The  system  hierarchy  is,  in  reality,  a network  in  which  data 
may  be  accepted  almost  anywhere  and  may  flow  in  any  direction. 
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y This  complicates  the  analysts'  task  of  developing  detailed  system 

model  specifications.  However,  model  and  simulation  development 
can  succeed  only  if  the  analysis  of  the  system  has  been  rigorous  and 
well  documented.  This  does  not  mean  that  the  model  specifications 
will  require  complete  detail  correspondence  between  model  com- 
ponents and  elements  in  the  system  hierarchy.  The  model  itself  will 
have  to  be  developed  on  the  basis  of  a number  of  assumptions.  It  will 
contain  many  formalizations  and  mathematical  idealizations,  and  it 
' will  be  designed  only  to  answer  questions  concerning  system  per- 

formance which  were  formalized  in  the  original  statement  of  need  or 
which  were  agreed  to  later,  during  analysis  investigations. 

4.1.2  Analysis  Documentation 
Requirements 

a.  The  end  result  of  the  system  analysis  investigation  should  be 
a detailed  model  functional  requirements  document  (see  Fig.  1)  which 
should  contain  the  following: 

(1)  The  original  statement  of  need  for  simulation  development  J 

(2)  Input-process -output  descriptions,  complete  with  diagrams 
depicting  all  system  elements  to  be  modelled.  These  descriptions 
must  accurately  represent  the  system's  interaction  with  its  environ- 
ment and  the  interactions  and  interrelationships  between  system  com- 
ponents. 

(3)  Events  and  event  sequences  over  the  system's  operations 
cycle,  with  detailed  information  on  combinations,  options,  timing, 
etc.  , to  provide  a basis  for  model  executive  logic  development 

(4)  Data  and  plans  for  model  testing  with  specific  requirements 
for  verification  and  validation  tests  which  should  be  performed  to 
determine  the  acceptability  of  any  proposed  model  assumptions, 
simplifications,  etc, 

. Within  the  input-process-output  descriptions,  careful  attention  must 

be  given  to  describing  all  parameters  as  to  data  source,  units, 
nominal  values,  role  in  system/subsystem  performance  evaluation, 
etc.  Moreover,  system  and  environmental  random  variables  must  be 
characterized  (statistics,  distributions,  etc.)  as  accurately  as 
possible, 
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b.  It  should  be  reemphasized  that  the  success  of  the  following 
model  and  simulation  development  stages  is  completely  dependent 
on  the  results  of  the  system  analysis  activity,  and  unless  the  analysis 
results  are  well  documented  in  the  model  functional  requirements 
document,  the  entire  project  could  easily  fail. 

4.  2 Implementation 

There  are  two  principal  activities  to  be  pursued  in  the  implementation 
phase  of  simulation  development:  model  development  and  simulation 
program  development.  Both  endeavors  are  highly  dependent  on  the 
results  of  system  analyses  and  interdependent  on  each  other.  Model 
development  involves  designing  and  developing  mathematical  models 
of  the  system  components'  functions  (processes),  organization  of  the 
models  into  modules  comparable  to  the  system  components  which  they 
must  emulate,  and  development  of  the  control  logic  required  to  inter- 
face the  modules  and  effect  the  system's  interactive  and  intra-active 
relationships  in  the  system  model.  Simulation  program  development 
is  the  activity  of  transforming  the  system  model  into  an  operational 
computer  program  which  is  functionally  equivalent  to  the  system  model. 
This  effort  involves  selecting  the  appropriate  computer  hardware 
(analog,  digital,  or  hybrid),  translating  detailed  model  descriptions 
into  a computer  program  (coding  in  the  selected  simulation  language, 
developing  analog  program  circuitry  diagrams,  etc. ),  and  then 
implementing  the  program.  Sections  4.  2.  1 and  4.  2.  2 discuss  model 
development  and  simulation  program  development  in  more  detail. 

4.  2.  1 Model  Development 

a.  The  model  functional  requirements  document  provides  a com- 
prehensive description  of  the  conceptual  system  model,  and  often  the 
document  will  completely  specify  many  of  the  math  models  to  be  used 
(the  system  and  subsystem  hierarchical  input-process -output  descrip- 
tions are,  themselves,  most  often,  models  of  the  system's  operation). 
However,  in  cases  where  analysis  results  do  not  present  specific 
models,  they  must  then  be  designed  in  the  model  development  stage. 

b.  The  principal  objective  of  model  development  is  to  establish 
computational  procedures  (equations,  algorithms,  logic,  etc. ) which 
will  serve  as  input -to -output  transformations  which  are  functionally 
equivalent  to  their  system/subsystem  process  counterparts.  Again, 
this  activity  should  be  constrained  by  the  model  functional  requirements 
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document  which  serves  as  a prescription  for  methodology  whereby  the 
original  simulation  requirements  may  be  satisfied. 

4.  2.1.1  Method  for  Model  Development 

a.  .The  specific  methodology  for  system  model  development  should 
be  pursued  as  follows: 

(1)  Organize  system  level  input  (environment,  stimuli,  etc.) 
information  from  the  system  analysis  investigation  for  the  purpose  of 
completely  characterizing  all  system  model  input  scenarios. 

(2)  Associate  the  output  requirements  specified  in  the  model 
functional  requirements  document  with  specific  input  scenarios. 

(3)  Based  on  the  system  process  description  as  discussed  under 
system  analysis,  design  the  system  model  executive  module  to  accept 
input  scenarios,  to  sequence  events  and  establish  action  timing,  to 
organize  subsystem  models  actions,  and  to  organize  subsystem  models 
responses  to  produce  the  system  model  responses  (outputs).  It  is  at 
this  level  that  the  system's  interaction  with  its  environment  is  modelled 
with  the  establishment  of  system  level  input  and  output  procedures. 

The  system-with- subsystem  relationships  are  also  reflected  in  the 
executive  module  by  sequencing  and  commanding  subsystem  model 
process  performance  and  by  receiving  and  organizing  subsystem  model 
responses. 

(4)  Once  the  system  model  executive  module  is  established, 
proceed  to  develop  subsystem  model  modules  for  processes  at  succes- 
sively lower  levels  in  the  system  hierarchy  descriptions  which  were 
established  during  system  analysis.  As  was  discussed  earlier,  each 
subsystem  is  in  itself  an  input- process -output  system.  Hence,  the 
methodology  for  developing  subsystem  model  modules  is  the  same  as 
that  for  developing  system  modules.  The  primary  considerations  here, 
again,  are  that  the  subsystem  modules  must  accept  input  from  the  system 
module  and/or  other  subsystem  modules,  they  must  process  informa- 
tion with  the  math  models  and  algorithms,  and  they  must  organize 
actions  of  lower  level  modules  and  utilize  their  results  in  establishing 
subsystem  model  responses. 

From  the  discussion  above,  it  should  be  obvious  that  the  system  and  sub- 
system model  interfaces  are  accomplished  by  the  input  and  output 
procedures. 
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b.  As  each  module  is  developed,  it  should  be  analytically  tested 
to  insure  that  it  will  perform  as  intended  on  data  from  the  system 
analysis  investigation  and  on  data  which  can  be  formulated  during  model 
development  to  demonstrate  that  performance  objectives  have  been 
satisfied.  Moreover,  as  modules  which  must  interact  with  each  other 
are  completed,  they  should  be  tested  collectively,  and  eventually  the 
composite  system  model  should  be  tested  against  specific  system  input 
scenarios. 

c.  The  top-down  structured  approach  to  system  model  develop- 
ment discussed  above  is  very  effective  in  two  respects:  It  completely 
defines  the  required  level  of  system  detail  to  be  modelled,  and  the 
resulting  detailed  model  descriptions  facilitate  the  subsequent  simula- 
tion program  development. 

4.  2.  1.  2 Model  Documentation  Requirement.  As  the  system  and  sub- 
system models  are  developed,  detailed  descriptions  of  them  should  be 
written  and  organized  into  a simulation  program  specification  document 
which  must  present: 

a.  Objectives 

b.  Input-process -output  descriptions 

c.  Events  and  event  sequences 

d.  Data  and  plans  for  program  testing 

This  document  must  be  specifically  oriented  toward  computer  implementa- 
tion of  a simulation  program  which  will  be  functionally  equivalent  to  the 
system  model.  In  particular,  the  quantity,  types,  and  formats  of  data 
(input  and  output)  must  be  described  in  detail.  Additionally,  any  further 
program  verification  and  validation  tests  which  should  be  performed  to 
judge  model  simplifications  and  assumptions  should  be  completely 
specified. 

4,  2.  2 Simulation  Program  Development 

Developing  a system  simulation  computer  program  is  analogous  to 
fabricating  the  system  itself  from  detailed  design  plans.  The 
primary  objective  for  this  stage  of  development  is  to  transform  the 
functional  system  descriptions  which  resulted  from  system  analysis 
and  model  development  into  an  operational  computer  program  which 
is  functionally  equivalent  to  the  system  model.  Once  its  performance 
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has  been  proven  acceptable,  the  simulation  program  will  be  a practical 
experimental  device  which  can  be  used  to  generate  data  predictive  of 
the  performance  capabilities  of  those  elements  of  the  system  which 
have  been  simulated. 

4.  2.  2.  1 Program  Development  Methodology 

a.  As  indicated  earlier,  simulation  program  development  involves 
three  primary  tasks: 

(1)  Analysis  of  model  descriptions  in  the  program  specification 
document,  to  determine  appropriate  computer  hardware  (analog, 
digital,  or  hybrid)  for  model  modules 

(2)  Selecting  appropriate  programming  techniques  (simulation 
language,  etc.  ) and  development  of  the  program  (coding,  analog 
circuitry  diagramming,  etc. ) 

(3)  Implementing,  debugging,  and  testing  the  program 

b.  Considering  each  program  development  task  in  turn,  based  on 
detailed  model  descriptions  and  simulation  specifications,  each  com- 
ponent (module)  of  the  system  model  should  be  examined  to  identify  the 
most  appropriate  computer  equipment  for  implementation.  Some  of 
the  factors  to  be  considered  in  making  the  selections  include: 

(1)  Proposed  use  of  the  program 

(2)  Number  and  types  of  runs  to  be  required 

(3)  Available  resources 

(4)  Level  of  model-to- system  detail  required 

(5)  Comparative  degree  of  implementation  difficulty 

(6)  Implementation  schedule  requirements 

(7)  Program  flexibility  requirements 

c.  After  selection  of  the  computer  hardware  to  be  used,  program 
development  should  be  pursued  following  top-down,  structured  approach 
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analogous  to  model  development.  The  program  executive  routine 
(most  often  the  main  program)  should  be  developed  (coded  in  an 
appropriate  simulation  language  or  diagrammed  for  analog  implementa- 
tion) to  accomplish  the  following: 

(1)  Accept  properly  formatted  system  input  scenarios  and  translate 
them  into  system/subsystem  action  requirements  (logical  options, 
input  or  stimuli  to  functions  and  subroutines,  etc.) 

(2)  Sequence  events  and  establish  action  timing 

(3)  Call  on  subroutines  to  perform  operations  functionally  equiva- 
lent to  subsystem  model  processes  on  their  input 

(4)  Organize  subroutine  results  (output)  and  combine  them  to 
establish  program  output 

(5)  Format  and  produce  program  results  (output) 

d.  It  is  not  mandatory  that  all  of  the  executive  functions  be  per- 
formed in  a single  routine.  Input  and  output  procedures,  for  example, 
may  be  established  in  separate  routines  which  have  the  capacity  for 
communicating  data  to  and  from  the  main  program  and/or  any  other 
subroutine.  Following  the  development  of  the  simulation  executive 
routine,  subroutines  should  be  developed  to  perform  functions  corre- 
sponding to  subsystem  model  processes  at  successively  lower  levels  in 
the  system  model  hierarchy.  Specifically  involved  in  subroutine 
development  is  the  translation  of  subsystem  model  processes  (math 
models,  stochastic  processes,  algorithms,  etc.  ) into  compute r- 
sensible  procedures  which  will  accept  appropriate  input  and  transform 
it  into  the  required  subsystem  model  response.  The  key  steps  to  be 
followed  are: 

(1)  Establish  machine -sensible  descriptions  for  program  elements 
(cards,  tapes,  diagrams,  etc.  ) 

(2)  Enter  the  program  elements  into  (on)  the  computer 

(3)  Translate  digital  program  routines  in  higher  level  languages 
by  computer  system  software  (compilers)  into  "machine  language" 

(often  relocatable  binary  code  elements) 
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(4)  Implement  digital  program  by  computer  system  software 
processing  (collection)  of  the  binary  elements  into  an  absolute 
executable  program 

The  correction  of  program  implementation  errors  and  repetition  of 
the  step  in  which  they  were  encountered  are  called  program  "debugging." 

e.  As  each  of  the  program  elements  is  completed,  it  should  be 
tested  with  sample  inputs  as  prescribed  in  the  simulation  requirements 
specification  document  and,  on  completion  and  collection  of  all  program 
elements,  the  simulation  program  should  be  tested  as  a whole  with 
more  comprehensive  input  scenarios.  The  purpose  of  these  tests  should 
be  to  completely  debug  and  check  out  the  program  and  to  determine 
whether  it  will  perform  as  intended  in  correspondence  with  the  system 
model. 

4.  2.  2.  2 Simulation  Program  Documentation.  During  model  and 
simulation  program  development,  a simulation  program  description 
document  should  be  prepared.  This  document  is  the  primary  source 
for  program  user  information.  Moreover,  it  presents  a single  source 
of  information  for  the  full  development  activity,  from  statement  of 
requirement  to  operational  program.  The  simulation  program 
description  document  should  present  at  least  the  following  information: 

a.  An  abstract  statement  of  purpose  and  function  (This  should 
reflect  the  original  statement  of  requirement  for  development  and 
should  provide  a general  description  of  the  input-process-output 
sequence. ) 

b.  Special  features  and  requirements  (program  limits,  restric- 
tions, computer  configuration,  language  requirements,  etc.) 

c.  Detailed  user  information  (all  input  requirements,  such  as 
files  and  variables  with  names,  formats,  units,  etc.  ; output  provided, 
such  as  sample  listings,  plots,  etc.;  and  job  control  requirements  for 
program  execution) 

d.  A detailed  program  methodology  section  to  present,  on  a 
routine -by- routine  basis,  a comprehensive  description  of  the  com- 
putational procedures  used  to  simulate  the  various  system  elements. 

This  section  should  also  contain  functional  flow  diagrams  for  each 
of  the  program  routines. 
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e.  Sample  runs  (input  and  output) 

f.  A preliminary  assessment  of  the  program  on  the  basis  of 
checkout  test  results 

In  addition  to  the  simulation  program  description  document,  a com- 
prehensive statement  of  program  verification  requirements  should 
be  established  prior  to  the  conclusion  of  the  implementation  stage. 
Also,  any  requirements  for  validation  testing  (to  demonstrate 
acceptability  of  assumptions,  simplifications,  etc.  in  implementa- 
tion) should  be  completely  documented. 

4.  3 Verification 


Once  a simulation  program  has  been  developed  and  implemented,  and 
prior  to  its  use  as  a system  evaluation  tool,  its  capacity  for  represent- 
ing its  counterpart  (the  physical  system)  must  be  established.  The 
simulation  verification  and  the  simulation  validation  stages  which 
follow  are  conducted  to  establish  program  credibility,  and  both 
activities  must  be  performed  before  adequate  confidence  in  the 
program's  performance  can  be  achieved.  Validation  demonstrates 
comparability  between  the  simulation  program  responses  (outputs) 
and  the  responses  from  the  modelled  system  for  equivalent  stimuli 
(inputs).  Verification  establishes  the  integrity  of  the  system  model 
with  respect  to  the  design  of  the  actual  system  and  establishes, 
analytically,  the  rectitude  of  the  program  with  respect  to  the  system 
model. 

4.  3.  1 Verification  Analysis  and  Testing.  Verification  is  a basic 
concern  throughout  the  simulation  development  cycle.  The  degree 
to  which  it  must  be  a separate  activity,  once  an  operational  simula- 
tion program  is  available,  depends  on  whether  the  program  was 
developed  specifically  to  satisfy  the  objectives  of  the  current  develop- 
ment or  whether  it  was  a preexisting  program  which  was  adapted 
to  satisfy  those  objectives.  If,  in  response  to  the  original  statement 
of  requirement,  the  program  has  evolved  through  the  previously 
discussed  simulation  development  stages,  the  major  part  of  the 
verification  investigation  will  already  have  been  accomplished,  and 
all  that  remains  is  to  present  the  appropriate  results  from  the  analysis, 
model  development,  and  implementation  stages  with  pertinent  test 
results  in  the  program  verification  report.  On  the  other  hand,  if  the 
program  was  obtained,  and  must  be  evaluated  for  its  applicability, 
its  verification  can  be  satisfactorily  accomplished  only  if  it  is 
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analyzed  in  parallel  with  the  physical  system,  following  the  methodology 
specified  in  the  system  analysis  and  model  requirements  definition 
development  stage  (Section  4. 1).  Regardless  of  the  program's  origin, 
however,  the  key  verification  activities  may  be  itemized  as  follows: 

a.  Ascertaining  the  degree  of  correspondence  (functional 
equivalence)  between  the  physical  system  and  the  system  model  and 
judging  that  correspondence  on  the  basis  of  what  the  resulting  simula- 
tion is  to  be  used  for  (reference  the  original  statement  of  requirement) 

b.  Ascertaining  the  rectitude  of  the  operational  simulation  com- 
puter program  with  respect  to  the  system  model 

c.  Insuring  that  the  program  is  syntactically  free  of  errors 

d.  Running  specific  test  cases  with  the  program  to  obtain  results 
that  can  be  carefully  checked  against  known  relationships  and  against 
manually  derived  results  from  math  models  in  the  system  model 

e.  Specifically  documenting  all  identifiable  simplifications  or 
idealizations  in  either  the  system  model  or  the  simulation  program 
which  must  be  further  evaluated  in  the  program  validation  stage  to 
follow 

4,  3.  2 Program  Verification 
Doc  umentati  on 


The  results  from  verification  analyses  must  be  carefully  examined  by 
simulation  personnel,  analysts,  project  directors  and  decision  makers 
alike  to  ascertain  whether  the  program  will  be  suitable  for  the 
prescribed  applications  (from  the  statement  of  requirement).  The 
material  to  support  the  evaluation  of  these  results  should  be  presented 
in  a simulation  verification  report  and  should  consist  of  at  least  the 
following: 

a.  Statement  cf  requirements  (adapted  and  expanded  from  the 
original  statement  of  requirement) 

b.  Parallel  functional  descriptions  of  the  system,  system  model, 
and  simulation  structures  (hierarchical  input-process-output  sequences) 
which  depict  cross -correspondence 
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c.  Results  from  specific  tests  which  have  been  conducted 

d.  Discussion  of  all  assumptions,  simplifications,  etc,  , which 
must  be  evaluated  by  specific  validation  testing 

From  the  evaluation  of  the  information  in  the  simulation  verification 
report,  a decision  must  be  made  to  either  reexamine  earlier  analyses 
and  make  revisions  if  the  program  is  inadequate,  or  to  proceed  with 
validation  testing. 

4. 4 Validation 

The  objective  of  simulation  validation  is  to  demonstrate  that  the 
responses  (outputs)  generated  by  a simulation  correspond,  within  an 
allowable  tolerance,  to  actual  system  performance  measurements  on 
the  key  parameters  (system  state  variables)  which  must  be  used  to 
evaluate  the  system.  Before  validation  investigations  are  conducted, 
the  following  three  criteria  must  be  satisfied: 

a.  System  performance  data  must  be  obtained  from  tests  which 
are  designed  to  establish  critical  values  for  the  system  state  variables. 

b.  All  environmental  conditions  which  directly  influence  system 
performance  should  be  measured  and  recorded  for  specification  of 
simulation  initial  conditions  (inputs). 

c.  The  simulation  to  be  validated  must  accommodate  input  speci- 
fication of  all  critical  test  initial  conditions. 

The  simulation  runs  must  be  constrained  to  the  known  system  con- 
ditions for  the  test(s)  to  which  they  are  to  be  compared;  otherwise, 
conclusions  drawn  from  validation  testing  may  be  completely  errone- 
ous. Special  care  must  be  taken  to  numerically  characterize  all  test 
environmental  conditions.  This  must  be  done  to  preclude  superior 
(inferior)  performance  by  the  simulation  which  could  be  caused  by  too 
favorable  (unfavorable)  initial  conditions.  Where  it  is  impossible  to 
quantify  some  of  the  test  conditions  exactly  (errors  in  observation, 
etc.  ),  the  quantities  should  be  depicted  distributionally,  and  Monte 
Carlo  techniques  should  be  used  to  select  the  corresponding  simulation 
initial  conditions. 
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4.  4.  1 Validation  Analysia  and  Testing 

a.  The  decision  as  to  which  kinds  of  procedures  should  be  used  to 
validate  a simulation  is  constrained  by  more  than  the  limitations  on 
testing  the  system.  The  kind  of  model  which  the  simulation  embodies 
must  also  be  taken  into  consideration.  Deterministic  models  which  do 
not  incorporate  random  environmental  effects  on  the  system  or  the 
random  effects  within  the  system  can  only  be  judged  subjectively, 
since  one  can,  at  best,  proceed  through  the  simulation  verification 
stage  to  ascertain  design  integrity  and  then  make  a judgment  concern- 
ing validity  by  examining  the  simulation  results  in  comparison  with 
the  results  from  a large  number  of  system  tests.  Dynamic  stochastic 
simulations  differ  from  purely  deterministic  simulations  in  that  they 
also  include  random  effects,  and  their  responses  can  be  viewed  ao 
random  variables.  Moreover,  the  methods  which  can  be  employed  to 
validate  this  class  of  simulations  are  well-established  statistical  pro- 
cedures for  comparing  random  variables. 

b.  Considering  the  responses  from  a dynamic  stochastic  simula- 
tion as  stochastic  processes  or  random  variables,  there  are  two  basic 
categories  to  be  treated: 

(1)  Singular  or  event-oriented  responses:  variables  for  which 
one  or  a discrete  number  of  values  are  generated  on  any  given  simu- 
lation run  or  system  test  {e,  g. , miss  distance,  event  time  occur- 
rences, etc.  ) 

(2)  Time  series  responses:  system  state  variables  which  are 
considered  to  be  time  dependent  functions  (e.  g. , missile  or  target 
positions,  velocities,  accelerations,  etc.) 

c.  The  statistical  procedures  for  comparing  system  and  simula- 
tion responses  which  are  from  the  first  category  above  are  straight- 
forward and  relatively  easy  to  apply.  There  are  a number  of  useful 
statistical  tests,  both  parametric  and  nonparametric,  which  can  be 
used  to  establish  confidence  in  comparability  between  simulation  and 
system  responses  of  this  type.  Time  series  responses  pose  more  of 
a problem.  Though  the  comparison  tests  for  variables  from  the  first 
class  are  still  applicable,  the  time  series  responses  will  have  to  be 
compared  discretely  at  selected  points  in  time  over  their  total 
duration. 
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d.  In  cases  where  system  test  data  are  too  limited  to  permit 
statistical  validation  comparison,  the  analyst  should  examine  as  much 
subjective  comparison  data  as  can  be  obtained.  The  most  frequently 
used  method  is  to  analyze  overlay  plots  of  the  system  test  and  simu- 
lated responses. 

e.  Appendix  A to  this  TOP  provides  a detailed  discussion  on  a 
number  of  commonly  used  statistical  tests  which  may  be  used  for 
simulation  validation. 

f.  A note  of  qualification  is  in  order  regarding  application  of  the 
procedures  for  validating  simulations.  In  a complete,  abstract  sense, 
simulation  validation  cannot  be  accomplished.  At  best,  one  can  infer 
simulation -system  correspondence  based  on  the  lack  of  sufficient 
contradictive  evidence  (e.  g. , in  statistics,  when  testing  a null 
hypothesis,  one  either  rejects  or  does  not  reject  the  hypothesis  based 
on  whether  the  computed  value  of  a test  statistic  is  judged  to  be  signif- 
icant). The  simulation  analyst  should,  therefore,  be  very  cautious 
when  claiming  validity. 

4.  4.  2 Program  Validation  Documentation 

Model  and  simulation  credibility  are  basic  concerns  throughout  all  five 
stages  of  simulation  development.  While  validation  test  results  alone 
cannot  completely  establish  program  credibility,  they  provide  a basis 
for  making  the  major  decision  to  utilize  the  simulation  for  system  per- 
formance capability  evaluation.  Because  of  their  impact  on  that  deci- 
sion, the  validation  test  results  should  be  presented  in  a formal  report 
which  completely  and  accurately  depicts  the  simulation's  performance 
capability.  The  following  information  should  be  included  in  such  a 
report: 


a.  Objectives.  System  performance  features  which  have  been 
exercised  (relatable  to  original  statement  of  requirement) 

b.  Test  conditions.  Complete  description  of  all  system  input 
(including  environmental  conditions)  and  its  correspondence  with  simu- 
lation input  and  initial  conditions  . 

c.  System  data  collection  procedures.  Methods  used  in  measuring 
system  input  and  response  data 
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d.  Methods  of  response  comparison.  Methods  used  for  system- 
simulation  response  comparison  including  response  (parameter) 
identification,  comparison  results  (statistics,  plots,  etc.) 

e.  Conclusions.  Complete  analysis  of  the  validation  test  results, 
including  statistical  significance  of  deviations,  identification  of 

inadr qua cie s , recommendations  for  program  improvement,  etc. 

4.  5 Applications 

a.  The  applications  stage  of  model  development  is  the  realization 
of  the  objectives  for  which  the  simulation  development  was  initiated.  It 
is  in  this  stage  that  the  verified  and  validated  simulation  program  is 
used  by  analysts  and  decision  makers  to  gain  insight  into  how  the  actual 
system  would  probably  perform  in  response  to  a wide  variety  of  condi- 
tions where  system  performance  demonstration  is  impractical.  The 
word  "probably " has  been  stressed  as  a reminder  that  conclusions 
regarding  system  performance  capability  which  are  based  on  simulation 
results  can  only  be  inferences,  with  their  attendant  statements  of  signi- 
ficance or  confidence. 

b.  There  are  many  simulation  applications  possibilities,  among 
which  are  the  following: 

(1)  Test  planning.  Simulation  results  often  identify  potential 
system  problems  which  should  be  examined  on  the  basis  of  specific 
system  tests. 

(2)  System  design.  Simulation  may  be  used  to  evaluate  the  impact 
of  alternative  design  plans  on  system  performance. 

(3)  Performance  prediction.  The  simulation  may  be  used  to  pre- 
dict system  performance  for  specific  conditions  under  which  the  system 
is  actually  to  be  tested  (safety  considerations,  test  adequacy,  etc.  ). 

(4)  Sensitivity  analysis.  Inputs  to  the  simulation  may  be  varied 
parametrically  to  identify  system  sensitivities. 

(5)  System  performance  evaluation.  Cn  the  basis  of  simulation 
results,  analysts  can  make  inferences  concerning  the  system's 
capability  to  meet  performance  objectives  (specifications,  require- 
ments, etc, ). 
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(6)  System  capability  assessment.  Where  impractical  or  impos- 
sible by  system  testing,  probable  system  performance  boundaries  can 
be  established  on  the  basis  of  simulation  results, 

4.  5.  1 Comments  on  Applications  Methodology 

Even  though  simulation  application  requirements  vary  widely,  it  is 
extremely  important  that  certain  basic  procedures  be  followed  in  con- 
ducting simulation  investigations.  The  key  ingredients  for  a successful, 
timely,  and  effective  simulation  study  are  as  follows: 

a.  Goals.  The  objectives  for  the  study  must  be  carefully  formalized 
and  judged  to  be  within  the  simulation's  domain  of  applicability. 

b.  Schedules  and  milestones.  There  must  be  complete  agreement 
among  analysts,  simulation  personnel,  and  decision  makers  on  what  can 
be  accomplished  and  in  what  time  frame. 

c.  Data  collection  and  presentation  of  results.  Well-organized 
procedures  must  be  established  for  collection  of  pertinent  system  test 
data  and  for  presenting  an  appropriate  interpretation  of  simulation 
results. 

4.  5.  2 Documentation  of 

Simulation  Applications 

4.  5.  2.  1 Simulation  Applications  Reports.  The  results  from  specific 
simulation  applications  investigations  should  be  documented  in  detail 
in  simulation  applications  reports  to  be  provided  to  analysts  and  deci- 
sion makers  for  making  system  assessments.  Such  reports  should 
contain  the  following  information: 

a.  Objectives.  A concise  review  of  what  the  study  was  supposed 
to  accomplish  with  a summary  assessment  of  accomplishments  and 
significance  of  results 

b.  Investigation  details.  A complete  description  of  the  investi- 
gation including  data  collection,  simulation  case  run  descriptions  and 
results  (outputs),  and  problems  encountered 

c.  Conclusions.  A comprehensive  assessment  of  the  significance 
of  the  results  from  the  investigation  in  an  interpreted  form,  to  facilitate 
use  of  the  results  by  decision  makers 
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4.  5.  2.  2 Final  Program  Documentation.  Throughout  the  five  simulation 
development  stages,  appropriate  documentation  should  be  maintained 
and  updated  to  reflect  any  changes  in  project  direction,  changes  in  model 
or  program  implementations,  key  milestone  accomplishments,  etc.  In 
addition,  at  the  conclusion  of  a specific  system  evaluation  project,  a 
final  report  should  be  prepared  to  reflect  project  progress  through  the 
development  cycle.  The  report  should  provide  a single  composite 
source  for  information  on  a simulation  program,  beginning  with  its 
inception  and  concluding  with  significant  system  evaluation  accom- 
plishments which  were  realized  from  its  use.  The  final  project 
document  should  be  organized  in  a format  similar  to  the  simulation 
program  description  document  (Section  4.  2.  2.  2),  with  complete  develop- 
ment details  presented  in  the  methodology  section  and  the  results  of  all 
simulation  case  studies  presented  in  the  applications  section.  The 
final  report  should  also  provide  a complete  assessment  of  project 
accomplishments,  along  with  all  relevant  management  data  (personnel, 
cost,  schedules,  etc.). 

5.  SUMMARY 

^he  discipline  of  modelling  and  simulation  is  one  which  can  be  employed 
very  effectively  to  support  the  analysis  and  evaluation  of  systems  from 
almost  any  field  of  endeavor.  However,  to  insure  an  effective  con- 
tribution, it  is  imperative  that  simulation  development  be  pursued  as  a 
five-stage  activity  with  the  objective  of  satisfying  applications  require- 
ments which  should  be  established  jointly  by  system  analysts,  system 
developers,  simulation  personnel,  and  project  managers.  The  five 
simulation  development  stages  are:  ^ 

a.  System  analysis  and  model  requirements  definition 

b.  Implementation 

c.  Verification 

d.  Validation 

e.  Applications 

The  development  stages,  while  often  iterative,  reflect  the  logical 
succession  of  activities  from  program  inception  through  system  evalua- 
tion. The  requirements  for  detailed  documentation  development  during 
each  of  the  five  stages  can  be  easily  satisfied  by  formally  describing 
investigation  activities,  and  are  essential  to  successful  project 
accomplishment. 
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Recommended  changes  to  this  publication  should  be  forwarded  to 
Commander , US  Army  Test  and  Evaluation  Command,  ATTN:  DP.STE-AD-M, 
Aberdeen  Proving  Ground,  MD  21005.  Technical  information  may  be 
obtained  from  the  preparing  activity:  Commander,  US  Army  White 
Sands  Missile  Range,  ATTN:  STEWS-TE-PR,  White  Sands  Missile  Range, 
NM  88002.  Additonal  copies  are  available  from  the  Defense  Docu- 
mentation Center,  Cameron  Station,  Alexandria,  VA  22314.  This 
document  is  identified  by  the  accession  number  (AD  No.)  printed  on 
the  first  page. 
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APPENDIX  A 

STATISTICAL  TESTS  AND  PROCEDURES  FOR 
SIMULATION  VALIDATION 


1.  The  Kolmogorov-Smirnov 

Distribution  Equivalence  Teata 


a.  The  Kolmogorov -Smirnov  one-aample  and  two-aample  teata 
(referred  to  aa  K-Sl  and  K-S2)  are  nonparametric  (i.  e. , "distribution- 
free")  teats  which  can  be  uaed  to  infer  equivalence  between  the  diatri- 
bution functiona  for  two  random  variable  a and  hence  that  they  belong  to 
the  aame  statistical  population.  Both  the  K-Sl  and  the  K-S2  teats 
involve  comparison  (at  a choaen  level  of  significance)  of  teat  statis- 
tics , computationally  based  on  the  maximum  deviation  between  two 
diatribution  functiona,  with  standard  table  values  which  are  based  on 
sample  sizes.  The  K-Sl  test  can  be  used  for  comparing  the  computed 
estimates  for  the  distribution  function  from  one  sample  with  a theoret- 
ical or  a previously  determined  distribution  function,  whereas  the 
K-S2  test  can  be  used  for  comparing  the  computed  estimates  for  the 
distribution  functions  from  two  samples  with  each  other.  The  K-Sl 
test  should  be  used  when  a variable  is  to  be  examined  against  a postu- 
lated diatribution  or  when  much  more  data  is  available  from  one 
source  (simulation  or  system  tests)  than  can  be  obtained  from  the  other, 
since  larger  samples  provide  a better  characterization  of  the  theoret- 
ical distribution  function. 


b.  Two  assumptions  must  be  satisfied  before  applying  the  K-S 
te  sts: 


(1)  That  the  sample  observations  are  independently  selected 

(2)  That  the  distribution  functions  being  compared  are  continuous 

c.  The  computational  procedures  for  conducting  a K-S2  test  are 
as  follows  (K-Sl  is  a special  case  of  K-S2): 

(1)  For  random  variables  Xg  (simulation  response)  and  XT  (sys- 
tem test  response),  establish  the  hypothesis 

Ho:  F (a)  = G (a)  for  all  a, 

S T 

f*- 

A-  1 


j 

i 

i 

I 

! 

I 
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where  F and  G are  the  distribution  functions  for  the  simulation  response 
and  the  system  test  response,  respectively. 

This  hypothesis  may  be  tested  against: 

(a)  All  alternatives,  i.  e. , F (a)  4 G (a) 


or 


(b)  The  alternative  that  one  of  the  distribution  functions  is  sto- 
chastically greater  than  the  other  (e.  g. , Ha:  P(X_  < a)  > P(X  < a)) 

S X 

(2)  Proceed  by  rank  ordering  the  combined  sample  of  N a m + n 
observations  Xgj,  Xg£ , ...  , Xgm  and  X^.j,  X<j-£  # «.•  * X-j-n , and 
determine  values  for  the  empirical  distribution  functions  F and  G 
corresponding  to  each  of  the  N combined  sample  values.  Then  compute 
the  differences  Fi  - Gi.  Gi  - Fi,  and  (Fi  - Gij  for  i a 1,  N. 
Example:  Assume  that  the  following  samples  have  been  collected  for 
the  random  variable  X from  simulation  and  system  tests. 

2.1,  XS3  a 0.9,  XS4  a 0.7, 


XT1  = 2.0,  XT2  a 1.9,  XT3  a 0.8,  XT4  a 2. 3. 

After  following  steps  1 and  2 above,  we  have 

Ho:  F (a)  a Gv  (a)  for  all  a,  against  say 


Ha:  F (a)  4 Gv  (a). 


and  the  rank  order  table  for  i,  i a 1,  8: 


r 


SI 


1.7,  X 


S2 
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i_ 

Sample 

Xi 

fxi 

Fi  - Gi 

Gi  - Fi 

Fi  - 

1 

S 

0.7 

1/4 

0/4 

1/4 

-1/4 

1/4 

2 

T 

0.8 

1/4 

1/4 

0 

0 

0 

3 

S 

0.9 

2/4 

1/4 

1/4 

-1/4 

1/4 

4 

S 

1.7 

3/4 

1/4 

2/4 

-2/4 

2/4 

5 

T 

1.9 

3/4 

2/4 

1/4 

-1/4 

1/4 

6 

T 

2.0 

3/4 

3/4 

0 

0 

0 

7 

S 

2.  1 

4/4 

3/4 

1/4 

-1/4 

P/4 

8 

T 

2.3 

4/4 

4/4 

0 

0 

0 

Note  that  both  F.  and  value*  are  determined  by  dividing  the  numbers 
of  values  from  their  respective  samples  which  are  leas  than  or  equal 
to  the  i4*1  value  in  the  combined  sample  by  the  appropriate  sample  size. 


(3)  Three  different  test  statistics  may  now  be  computed  as  follows: 
(a)  D, 


1 


(b)  D,  * 


(C)  D3  a 


— — max  (Gi  - Fi) 
a 


mn 

d 

mn 


max  (Fi  - Gi) 
max  ( I Fi  -Gil) 


where  d is  the  greatest  common  divisor  of  m and  n. 

In  the  example  above,  the  null  hypothesis  is  to  be  tested  against  all 
alternatives,  so  we  would  be  interested  in  only.  For  the  example, 


D3  = (lUll  max  ( |Fi  - Gif ) * 4 


(!) 


2. 


This  value  may  now  be  compared,  at  say  a level  of  significance  corre- 
sponding to  a type  I error  of  size  a = 0.  1,  to  the  value  V (0.  1,  4,  4) 
from  the  K-S  tables  (Reference  2).  (Note:  The  constant  V(a,  m,  n) 
satisfies  the  relation  P[D  2 P (a,  m,  n)j  = a.)  From  the  tables,  we 
find  that  P(X  2 2)  > 0,  1,  and  we  would  not  reject  Ho. 

(4)  In  general,  reject  Ho  at  the  a level  of  significance  in  favor  of 


(a)  Ha:  Fv  (a)  4 G. 

A „ 


if  D3  > P{a,  m,  n), 


(a) 
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(b)  Ha:  Fy  (a)  < Gv  (a) 
XS  XT 


if  > Via,  m,  n). 


(c)  Ha:  F (a)  > Gv  (a) 
XS  XT 


if  D2  > Via,  m,  n). 


d.  The  quantity  a was  introduced  in  the  previous  discussion  and 
called  the  size  of  the  type  I error,  or  error  of  the  first  kind,  for  the 
test.  More  descriptively,  a is  the  probability  an  investigator  will  allow 
for  rejecting  a valid  hypothesis.  A second  error,  the  type  II  error,  or 
error  of  the  second  kind,  of  size  (3  must  also  be  considered  when 
testing  hypotheses.  Its  value  for  the  test  is  the  probability  of  not 
rejecting  a false  hypothesis.  This  quantity  is  often  called  the  analyst's 
risk,  p is  usually  depicted  as  operating  characteristics  curves 
obtained  by  plotting  this  probability  as  a function  of  significance  level 
and  sample  size.  Oven  (1)  presents  a formula  for  determining  sample 
size  for  the  K-Sl  test  to  attain  a given  power.  For  small  samples, 
the  K-S  testa  are  more  powerful  than  the  Wilcoxon  test  which  is  dis- 
cussed in  the  next  section.  For  large  samples,  however,  the  reverse 
is  true.  Hollander  and  Wolfe  (2)  provide  an  excellent  description  of  the 
K-S2  test,  and  they  also  include  a discussion  on  large  sample  approxi- 
mation. For  a more  detailed  discussion  on  the  efficiency  of  K-S 
testing,  see  Bradley  (3). 


2.  The  Wilcoxon  Rank  Sum  Test.  The  nonparametric  Wilcoxon  rank 
sum  test  is  applicable  for  comparing  the  locations  (medians)  from  two 
random  samples.  The  test  hypothesis  is  that  the  two  populations  from 
which  the  samples  were  drawn  have  the  same  location.  (For  a detailed 
discussion  on  the  Wilcoxon  rank  sum  test,  see  Reference  2. ) 


! 

t 


1.  Owen,  Donald  B.  (1962)  Handbook  of  Statistical  Tables.  Addison- 
Wesley  Publishing  Company,  Inc.  (15.6,  16.2) 

2.  Hollander,  Miles  and  Wolfe,  Douglas  A.  (1973)  Nonparametric 
Statistical  Methods.  John  Wiley  and  Sons,  Inc. 

3.  Bradley,  James  V.  (1968)  Distribution-Free  Statistical  Tests, 
Prentice -Hall  International,  Inc, 
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a.  The  assumptions  which  must  be  satisfied  are  as  follows: 

(1)  The  observations  are  mutually  independent. 

(2)  The  observations  in  each  sample  are  obtained  from  the  same 
continuous  population. 

b.  The  computational  procedures  for  conducting  the  Wilcoxon 
test  follow. 

(1)  Define  the  test  hypothesis  in  terms  of  D,  the  difference  in  the 
respective  population  locations,  to  be  Ho:  D = 0. 

The  null  hypothesis  may  be  tested  against: 

(a)  All  alternatives,  i.e.,  D 4 0, 

(b)  The  alternative  that  D > 0 , 


or 

(c)  The  alternative  that  D < 0. 

(2)  Proceed,  as  with  the  K-S2  technique,  by  rank  ordering  the 
N = m + n observations  Xg^  , Xg^  , . . . , Xg^  and  Xj  , . . . , X-p 
from  least  to  greatest,  and  assign  rank  values  Ri,  i = 1 , 2 , . . . , n, 
to  coincide  with  the  rank  of  the  X,^  in  the  ordering. 

(3)  Compute  the  test  statistic  W as  follows: 

n 

"i 

i = l 

(4)  Compare  the  test  statistic  W,  at  a desired  significance  level, 
with  the  table  value  (Reference  2)  forW(a,  m,  n),  where  PfW  5 (K(a,  m,  n)] 
= a. 


Based  on  the  result  of  this  comparison,  reject  Ho  at  the  a level  of 
significance  in  favor  of 

(a)  Ha:  D 4 0 

if  W > 0/(o2,  m,  n)  orW  < ( n(m  + n + 1)  - m,  a)]  » 

where  3 3 §' 
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if  W 2 W(a,  m,  n), 
(c)  Ha:  D < 0 


if  W s [n(m  + n + 1)  - W(a,  m,  n)] . 

Example:  In  the  example  which  was  used  *o  illustrate  the  K-S2  test, 
we  could  assign  rank  values  to  the  test  ob  ervations  as  follows: 


1 2.0  6 

2 1.9  5 

3 0.8  2 

4 2.  3 8 


and  then 

Ra  * 21  . 

i = 1 

For  a hypothesis  Ho  that  there  is  no  location  difference,  and  for  a * 0.  1 
0/(0.  1,  4,  4)  = 23, 


[n(m  + n + 

1) 

- 0/(0.  1,  m,  n)]  = 

36 

- 23  * 

13, 

[n(m  + n + 

1) 

- 0/(0.  05,  m,  n)  ] = 

36 

- 24  = 

12, 

and 


0/(0.  05,  4,  4)  s 24. 

Then  for  alternative  hypothesis  (4)(a)  defined  above,  since 

21  i 0/(0.  05,  4,  4)  » 24 


and 

21  i [4(4  + 4 + 1)  - 0/(0. 05,  4,  4)]  & 12 
we  do  not  reject  Ho  . 
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V*-  Similarly,  for  hypotheses  (4)(b)  and  (4)(c),  we  aee  that 

21  i W<0.  1,  4,  4)  = 23 


21  i [4(4  + 4 4-  1)  - (i/(0.  1,  4,  4)1  = 13 
and  we  would  reject  Ho  in  neither  case. 

c.  As  indicated  earlier,  the  Wilcoxon  test  is  stronger  than  the 
K-S  tests  when  the  sample  size  is  large.  There  are  two  considerations 
which  should  be  mentioned,  however.  First,  the  Wilcoxon  test  is  not 
valid  if  the  empirical  distribution  functions  from  the  two  sample  sources 
differ  significantly  in  form.  Second,  while  it  is  possible  to  apply  this 
test  to  smaller  samples,  as  the  example  illustrates,  the  K-S  tests  are 
still  to  be  preferred  for  small  samples.  Moreover,  the  K-S  tests 
ascertain  distributional  equivalence,  as  opposed  to  just  location  equiv- 
alence. For  a more  comprehensive  discussion  on  the  strength  of  the 
Wilcoxon  test,  refer  to  Bradley  (3). 

3.  Parametric  Comparison  Methods.  There  are  a number  of  classical 
statistical  techniques  which  are  applicable  for  comparing  two  random 
variables,  and  while  they  are  usually  somewhat  stronger  than  nonpara- 
metric  methods,  they  are  often  more  restrictive  in  that  they  are  only 
applicable  when  comparing  specific  parameters  (usually  distributional 
moments).  The  distribution  of  the  population  is  usually  known  and  of  a 
classical  statistical  form  (e.  g. , normal.  Student's  t,  Chi  Square,  etc.). 
There  are,  however,  a number  of  simulation  validation  parameters  to 
which  the  techniques  of  this  section  can  be  applied  with  the  objective  of 
increasing  confidence  in  the  simulations'  performance.  Since  the 
classical  procedures  which  will  be  referenced  in  this  section  are,  in 
general,  familiar  to  the  simulation  community,  they  will  not  be  expanded 
in  detail  as  were  the  K-S  and  Wilcoxon  tests.  Rather,  their  applications 
methodology  will  be  discussed  in  general  in  the  following  subsections. 

a.  The  Chi-Square  Goodness-of-Fit  Test 


(1)  The  Chi-Square  goodness-of-fit  test  is  applicable  for  testing 
the  hypothesis  that  a specific  sample  has  been  drawn  from  a known 
(usually  normal)  distribution.  Its  primary  advantage  over  purely 
parametric  methods  is  that  it  is  a test  for  complete  distributional 
equivalence.  The  assumptions  which  must  be  satisfied  before  applying 
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this  test  are  that  each  of  the  observations  in  the  sample  are  mutually 
independent,  and  that  they  are  drawn  from  the  same  population. 


(2)  The  application  of  this  procedure  involves  dividing  the  total 
range  of  the  sample  into  k classes  and  computing  a value  for  the  test 
statistic  X2  based  on  the  difference  between  the  observed  and  expected 
class  frequencies.  The  test  statistic  X2  may  then  be  compared  to  table 
values  for  a theoretical  Chi-square  distribution  with  k - 1 degrees  of 
freedom. 


(3)  The  primary  disadvantage  of  the  Chi-square  goodness -of-fit 
test  is  that  it  requires  sample  sizes  which  are  much  larger  than  the 
corresponding  distribution-free  K-S  techniques.  Bendat  and  Piersol 

(4)  provide  an  excellent  discussion  of  the  Chi-square  goodness-of-fit 
test  and  establish  criteria  for  determination  of  sample  size  and  of  the 
number  of  subintervals  to  use. 


b.  The  Normal  and  Student  t Teats 

(1)  These  tests  are  equally  appropriate  for  testing  the  hypothesis 
that  there  is  no  difference  between  two  populations  for  a given  distri- 
butional parameter  (e.  g.  , mean)  or  the  hypothesis  that  a particular 
sample  has  been  drawn  from  a theoretical  distribution  with  the  same 
value  for  that  parameter.  The  basic  difference  between  the  tests  is 
that  the  Student  t test  should  be  used  when  the  sample  size  is  less  than 
30  observations.  The  assumptions  which  must  be  satisfied  before  the 
test  can  be  applied  are  as  follows: 

fa)  The  test  statistic  must  be  normally  distributed  (student  t dis- 
tributed for  the  t test). 


(b)  The  sample  observations  must  be  mutually  independent. 

(2)  Procedurally,  when  applying  the  Normal  test  for  a given  popu- 
lation parameter  with  estimated  value  (commuted  from  the  sample), 

and  with  hypotl  asized  valu»  P , one  computes  the  statistic 

H 


?„ 

o 


H 


4.  Bendat,  fihus,  an*: 
Analysis  of  Random  Da:; 


•so,  Allan 


( 1966),  Measurement  and 


'ilev  and  Sons,  Inc. 
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where  Op  is  the  value  of  the  standard  deviation  for  that  parameter. 
The  statistic  Z is  then  compared  at  an  a level  of  significance  with 
tabulated  values  for  the  standardized  normal  variate  Za.  If  it  is 
larger,  the  test  hypothesis  should  be  rejected. 

(3)  The  procedure  for  applying  the  Student  t test  is  the  same. 
However,  the  statistic 


is  compared  to  values  from  a standard  t distribution  table. 


(4)  As  mentioned  above,  when  the  sample  size  is  less  than  30 
observations,  the  Student  t test  should  be  preferred  over  the  normal. 
For  cases  where  use  of  these  tests  is  applicable,  such  as  tests  for 
means  (M),  we  can  use  techniques  from  parametric  estimation  to 
establish  an  appropriate  sample  size.  Specifically,  we  choose  X units 
of  allowable  error  in  estimating  the  parameter  and  an  associated 
confidence  level  1 - «,  and  then  solve  for  n in  the  relationship 

za/2  • SA/IT=  X, 


where  Za/2  **  va*ue  °*  the  standardized  Normal  or  t variate,  as 
appropriate,  at  the  a/2  level  of  significance,  and  S is  the  sample 
standard  deviation  for  the  test  parameter. 


4.  Comparison  Procedures  for  Minimal  Sample  Sizes.  While  the 
modeller  and  the  systems  analyst  would  like  to  develop  the  highest 
level  of  confidence  possible  in  a simulation,  it  is  often  impossible  to 
perform  complete  rigorous  statistical  tests,  due  to  the  lack  of  suffi- 
cient test  data.  When  this  is  the  case,  one  can  only  obtain  enough 
simulation  data  to  distributionally  characterize  the  parameters  in 
question  and  then  comment  on  comparability  of  a test  result  based  on 
whether  it  falls  within  desired  confidence  limits.  The  following  pro- 
cedures are  the  more  commonly  used: 


a.  Plot  the  test  response  over  the  simulation  response  (with 
confidence  bounds  or  response  limits  if  the  simulation  is  a Monte 
Carlo  simulation). 


i 


t 


I 
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b.  Analyse  the  differences  between  the  test  response  and  the 
expected  response  (from  analysis  of  simulation  results) 

c.  Compute  time -averaged  integral  square  error  values  or  the 
variance  equivalent  between  the  test  response  and  the  response  pre- 
dicted by  the  simulation 

5.  Comments 


a.  The  methods  which  have  been  discussed  for  validation  do  not 
constitute  a completely  exhaustive  set  of  simulation  validation  pro- 
cedures. There  are  numerous  methods  which  could  be  employed;  how- 
ever, the  collection  of  methods  which  has  been  discussed  here  is  cer- 
tainly considered  appropriate  for  qualifying  simulations  as  instru- 
ments for  use  in  assessing  system  performance  capabilities. 

b.  Every  effort  must  be  made  to  collect  adequate  data  samples  so 
that  some  of  the  more  rigorous  statistical  comparison  techniques  can 
be  employed.  The  preferred  validation  tests  procedure  should  be 
supplemented  by  the  conduct  of  some  of  the  alternative  procedures 
(comparison  plots,  difference  analyses,  testing  of  hypotheses  about 
additional  distributional  statistics,  etc. ) so  that  the  highest  possible 
level  of  confidence  may  be  achieved. 
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